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Report  No.  97-105  March  6,  1997 

MEMORANDUM  FOR  UNDER  SECRETARY  OF  DEFENSE  FOR 

ACQUISITION  AND  TECHNOLOGY 
ASSISTANT  SECRETARY  OF  THE  AIR  FORCE 

(FINANCIAL  MANAGEMENT  AND  COMPTROLLER) 

SUBJECT:  Audit  of  Lessons  Learned  on  the  B-2  Training  System 


Introduction 

We  are  providing  this  report  for  your  information  and  use.  Our  review  of  the 
B-2  training  system  was  conducted  as  part  of  the  audit  of  "Requirements 
Planning,  Development,  Test  and  Evaluation,  and  Impact  on  Readiness  of 
Training  Simulators,"  Project  No.  5AB-0070.  We  will  issue  a  separate  report 
on  training  simulators  in  the  spring  of  1997.  The  B-2  Bomber  program 
experienced  significant  changes  in  its  requirements  that  affected  the  acquisition 
of  the  training  systems.  Because  of  the  mature  stage  of  the  B-2  program,  this 
report  provides  lessons  learned  to  improve  future  acquisitions  oif  training 
simulators  and  devices  instead  of  recommendations  related  to  the  B-2  program. 


Audit  Results 

We  identified  four  matters  that  may  provide  lessons  for  future  acquisitions, 
which  are  as  follows: 

o  The  B-2  Systems  Program  Office  bought  more  training  devices  than 
needed  to  support  21  air  vehicles. 

0  The  Air  Force  did  not  revalidate  the  need  for  the  various  components 
of  the  training  system  as  it  reduced  the  quantity  of  air  vehicles  that  it  intended  to 
buy. 


0  The  Air  Force  relied  upon  the  air  vehicle  prime  contractor  for  many 
training  devices  rather  than  separately  contracting  for  the  equipment. 

0  The  Air  Force  increased  the  cost  of  the  B-2  training  system  by 
$578.3  million  to  maintain  concurrency  between  the  training  system  and  the 
three  B-2  air  vehicle  configurations. 


Audit  Objectives 

The  objective  of  the  audit  was  to  determine  whether  requirements  planning, 
development,  test  and  evaluation,  and  the  potential  impact  on  readiness  of 
training  simulators  and  devices  were  adequately  considered  in  the  acquisition 
process.  The  B-2  program  was  1  of  30  programs  and  projects  included  in  our 
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review.  Specifically,  we  determined  whether  economic  analyses  were 
considered  as  part  of  the  decision  process,  whether  test  and  evaluation 
requirements  were  incorporated  into  the  Test  and  Evaluation  Master  Plan,  and 
whether  requirements  were  adequately  supported  and  documented  before 
development.  In  addition,  we  evaluated  the  impact  of  training  simulators  and 
devices  on  the  operational  readiness  of  the  Military  Departments.  The  final 
objective  was  to  evaluate  applicable  management  controls.  Because  Inspector 
Gener^,  DoD,  Report  No.  96-028,  "Implementation  of  the  DoD  Management 
Control  Program  for  Major  Defense  Acquisition  Programs,"  November  28, 
1995,  adequately  addresses  management  controls  for  the  B-2  program,  we  did 
not  review  applicable  management  controls.  See  Enclosure  1  for  a  discussion  of 
the  audit  scope  and  methodology.  See  Enclosure  2  for  a  summary  of  prior 
coverage  related  to  the  audit  objectives. 


Audit  Background 

The  B-2  Program.  The  B-2  Advanced  Technology  Bomber  is  an  all- wing, 
two-member  crew  aircraft  that  exploits  breakthroughs  in  low  observable 
technology  to  achieve  a  vehicle  signature  that  will  allow  penetration  of  current 
and  postulated  enemy  defenses.  In  the  early  1980's,  the  Air  Force  planned  and 
budgeted  to  procure  132  B-2  air  vehicles.  In  May  1990,  the  Secretary  of 
Defense  major  aircraft  review  prompted  a  major  restructure  of  the  B-2  program. 
As  a  result  of  the  review,  the  Air  Force  reduced  the  total  number  of  B-2  air 
vehicles  to  be  procured  from  132  to  75.  The  B-2  program  was  affected  again  in 
January  1992,  when  the  FY  1993  President's  Budget  reduced  the  total  air 
vehicle  procurement  from  75  to  20.  In  1996,  the  Air  Force  decided  to  upgrade 
the  B-2  flight  test  vehicle.  Air  Vehicle  1,  to  operational  status.  Therefore,  a 
total  of  21  air  vehicles  will  be  fully  operational.  Also,  in  response  to  a 
changing  threat  environment,  the  focus  of  the  B-2  mission  shifted  from  a 
strategic  nuclear  role  to  that  of  a  conventional,  deployable  bomber.  The  two 
reductions  in  air  vehicle  acquisition  quantities,  when  combined  with  three  block 
configurations  for  only  21  aircraft,  provide  the  backdrop  for  the  lessons  learned 
for  the  training  system  acquisition. 

The  B-2  Training  System.  The  B-2  training  system  is  an  integrated  system 
designed  to  support  air  crew  and  maintenance  training.  To  support  the  B-2 
program,  the  Air  Combat  Command  required  that  the  B-2  training  system 
maintain  concurrency  with  the  B-2  air  vehicle  configurations  and  be  delivered 
before  deployment  of  the  first  air  vehicle.  Also,  the  B-2  air  vehicles  would  not 
be  used  for  training  purposes. 

The  509th  Bomb  Wing  conducts  both  air  crew  and  maintenance  training  for  the 
B-2  program  at  Whiteman  Air  Force  Base,  Missouri,  the  main  location  of  the 
B-2  air  vehicles.  Representatives  of  the  509th  Bomb  Wing  expressed  overall 
satisfaction  with  the  B-2  training  system.  The  B-2  training  system  is  composed 
of  various  air  crew  and  maintenance  training  simulators  and  devices.  Refer  to 
Enclosure  3  for  a  description  of  each  component  of  the  B-2  training  system. 
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The  B-2  Training  System  Procurement.  In  1987,  the  Air  Force  contracted 
with  Northrop  Grumman  Cop)oration  (Northrop),  the  prime  contractor  for  the 
B-2  air  vehicle,  for  the  training  equipment  ne^ed  to  support  132  B-2  air 
vehicles.  Northrop  was  also  responsible  for  integration  of  each  of  the 
maintenance  and  air  crew  training  simulators  and  devices  into  a  single  training 
system.  Northrop  was  responsible  for  the  contract  and  subsequent  modifications 
for  the  development  and  production  of  the  following  components: 

o  Cockpit  Procedures  Trainers, 

o  the  Computerized  Maintenance  Training  System, 

o  the  Crew  Escape  Systems  Maintenance  Trainer, 

o  the  Flight  Control  System  Maintenance  Trainer, 

o  the  Weapons  Loading  Trainer, 

o  the  Weapon  Systems  Training  Aids,  and 

o  associated  courseware  and  support  systems. 

In  1985,  the  Air  Force  signed  a  separate  $133.5  million  contract  with 
Hughes-Link  (formerly  the  Singer  Company)  to  develop  and  produce  the 
Weapon  Systems  Trainers,  Mission  Trainers,  and  associated  mission  generation, 
debriefing,  and  support  systems.  The  estimated  cost  to  completion  of  that 
contract  has  now  increased  to  $548  million,  exclusive  of  the  cost  of  block 
upgrades. 


Discussion 

Table  1  shows  that  the  cost  of  the  B-2  training  system  is  much  greater  than  for 
other  air  vehicle  training  systems.  The  investment  cost  per  student  for  the  B-2 
training  system  with  21  B-2  air  vehicles  is  $1,533,613.45. 

Table  1.  Cost  of  Ground  Based  Training  Systems  and  the 
Number  of  Air  Vehicles  Supported 


Number  of  Maintenance  Trainers 

Number  of  Air  Crew  Trainers 

Total  Cost  of  the  Training  System 
(millions)^ 

Number  of  Air  Vehicles 

^F-22  cost  is  a  pre-production  estimate 


B-2 

F-22 

F-15E 

45 

241 

8 

15 

52 

4 

$1,460 

$748 

$271 

21 

438 

203 

3 


WWle  no  air  vehicle  directly  compares  to  the  B-2,  the  cost  of  procuring  the  B-2 

suDoUrted^^^TlU  A-  expensive  for  the  number  of  air  vehicles 

sup^rt^.  The  Air  For^  may  have  put  funds  to  better  use  if  management  had 
used  options  to  contract  for  the  trmning  system  and  revalidated  the  need  for  the 
v^ous  components  of  the  training  system.  The  Air  Force  may  have  also 
r^uced  costs  by  separately  contracting  for  the  training  equipment^rather  thS 
relying  on  the  pnme  contractor  for  most  of  the  training  sySem  coZonentT 
Fm^ly,  a  formal  process  to  determine  the  stability  o/  the  B-2  baseline 

Number  of  Trai^g  Simulators  and  Devices.  As  exhibited  in  Table  2  data 

No-  26096015,  "Transition  of 
Kuri°  Bomb  Wing,  Whiteman  Air  Force  Sse 

Missoun  Apnl  1,  1996,  suggest  that  the  Air  Force  bought  significantly  more 
frmmng  devices  and  simulators  than  needed  to  support  21  air  vehicles.  The 
Air  Force  may  have  been  able  to  acquire  fewer  training  simulators  and  devices 
had  It  used  contract  options  with  Northrop. 

Table  2.  Rate  of  Use  Based  on  Contractor 
Logistics  Support 
(FY  1995) 


T,  ■  .  o-  ,  Bate  of  Use 

- Training  Simulator  or  Device _  (nercenti 

Weapon  System  Trainer  5j 

Mission  Trainer  22 

Computerized  Maintenance  Training  System  21* 

Weapon  Systems  Training  Aid  14* 

Coc%it  Procedures  Trainer  17* 

Weapons  Loading  Trainer  54* 

*  Rate  does  not  include  September  1995  use  because  data  were  not 
available  at  the  time  of  audit  field  work. 

the  quantity  of  training  simulators  and  devices  required  on 
t  e  total  numbCT  of  people  to  be  trained  and  the  number  of  people  to  be  trained 

f  of  people  to  be  trained  for  the  B-2  changed 

sigmficMtly  following  reducfaons  m  the  number  of  B-2  air  vehicles  to  be 
procured.  When  the  quantity  of  B-2  air  vehicles  was  132,  the  number  of  S 

f  50o'”®Fnr®9 1  R  9^  ^ ®  ®”^*sted  personnel  was 

4,500.  For  21  B-2  ^  vehicles,  the  number  of  persons  to  be  trained  is  52  air 

crew  members  and  900  enlisted  personnel.  The  numbers  are  for  support  of 

132  air  vehicles  and  21  air  vehicles,  respectively,  and  do  not  represent 
trmmng.  The  number  of  people  to  be  trained  per  month  was 
derived  from  the  number  of  air  vehicles  to  be  delivered  per  month  When  the 

numter  of.B-2  air  vehiclei^lo  be  aSquM 
132  to  75,  the  rate  of  air  vehicles  delivered  per  month  remained  the  same 
Consequently,  the  Air  Force  determined  that  it  would  need  to  train  the  same 
number  of  people  per  month;  that  is,  the  initial  training  would  occur  over  fewer 


total  months,  but  the  Air  Force  would  train  the  same  number  of  students  per 
month.  Therefore,  the  Air  Force  assumed  that  the  quantity  of  training 
simulators  and  devices  required  to  support  the  B-2  program  would  remain 
unchanged. 

By  the  time  the  Air  Force  reduced  the  quantity  of  B-2  air  vehicles  to  be 
acquired  to  20,  the  training  devices  to  support  132  air  vehicles  had  already  been 
produced.  The  Air  Force  therefore  did  not  have  the  flexibility  to  reduce  the 
number  of  training  components  acquired  to  match  the  change  in  the  planned 
quantity  of  air  vehicles. 

The  Federal  Acquisition  Regulation  subpart  17.2,  "Options,"  defines  option  as 
"a  unilateral  right  in  a  contract  by  which,  for  a  specified  time,  the  Government 
may  elect  to  purchase  additional  supplies  or  services  called  for  by  the 
contract .  .  .  ."  The  Federal  Acquisition  Regulation  subpart  17.2  allows  the  use 
of  options  in  contracts  when  the  use  of  options  is  in  the  best  interest  of  the 
Government  and  when  the  option  does  not  represent  firm  requirements  for 
which  funds  are  available.  Because  only  11  air  vehicles  were  on  contract  when 
the  Air  Force  contracted  with  Northrop  for  all  the  equipment  needed  to  support 
132  air  vehicles,  the  Air  Force  had  a  firm  requirement  to  support  only  11  air 
vehicles.  By  contracting  for  an  initial  quantity  of  training  simulators  and 
devices  with  options  for  additional  quantities,  the  Air  Force  would  have  had  the 
flexibility  to  buy  fewer  training  simulators  and  devices  initially,  and  then 
increase  the  amount  in  increments  when  the  quantity  of  B-2  air  vehicles  to  be 
acquired  changed.  For  example,  the  use  of  options  on  the  contract  with 
Hughes-Link  allowed  the  Air  Force  to  acquire  only  three  of  the  eight  planned 
Weapons  System  Trainers. 

Cost-Effectiveness  of  the  Training  Media.  The  Air  Force  spent  about 
$578.3  million  to  upgrade  the  B-2  training  system  to  maintain  concurrency  with 
an  evolving  air  vehicle.  Considering  the  decrease  in  the  number  of  personnel  to 
be  trained,  the  decision  to  upgrade  all  the  training  devices  might  not  have  been 
the  most  cost-effective  solution  for  B-2  training.  The  Air  Force  may  have  been 
able  to  make  better  use  of  funds  had  it  reevaluated  the  training  media  mix 
before  committing  funds  to  upgrading  the  training  devices. 

Selecting  Training  Media.  The  Air  Force  applied  Instructional  Systems 
Development  procedures  to  both  air  crew  and  maintenance  training  to  identify 
design  requirements  and  to  develop  and  update  lessons  for  training  systeins. 
The  Instructional  Systems  Development  process  is  a  framework  to  identify 
training  requirements  for  the  design  and  development  of  materials  and  devices. 
The  process  provides  a  systematic  program  for  development  of  the  entire 
training  system  and  is  used  to  evaluate  cost-effectiveness  of  alternate 
approaches.  Part  of  the  Instructional  Systems  Development  process  involves  the 
selection  of  media.  Media  are  the  instruments  or  materials  used  to  communicate 
information.  Examples  of  media  range  from  classroom  instruction  to  training 
devices  and  high  fidelity  simulators.  Appropriate  media  selection  ensures  that 
the  information  to  be  learned  is  presented  in  the  most  effective  and  efficient 
means  possible. 
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The  training  media  for  the  B-2  training  system  are  composed  of  courseware  and 
training  simulators  and  devices.  The  Air  Force  selected  the  final  training  media 
based  on  the  Training  Media  Plan  prepared  by  Northrop  in  1984.  The  Training 
Media  Plan  analyzed  the  cost-effectiveness  of  the  candidate  training  media  using 
mathematical  models  based  on  the  life-cycle  cost  (LCC)  of  the  training  system 
and  the  number  of  personnel  to  be  trained.  The  number  of  personnel  to  be 
trained  was  derived  from  the  original  132  B-2  air  vehicles  that  were  to  be 
procured. 

Effects  of  Changing  Requirements.  If  the  basis  of  the  training  media 
changes,  so  does  the  cost-effectiveness.  For  example,  the  operating  costs  of 
traditional  media  such  as  classrooms,  on-the-job  training,  and  self-study  are 
highly  dependent  on  the  number  of  students,  or  what  the  Instructional  Systems 
Development  program  calls  the  "trained  personnel  requirement."  If  the  number 
of  personnel  to  be  trained  decreases  substantially,  the  total  LCC  of  traditional 
media  would  also  decrease.  The  change  in  the  LCC  can  reasonably  be  expected 
to  have  an  effect  on  the  relative  LCC  of  the  elements  of  the  training  system, 
which  were  used  in  evaluating  the  cost-effectiveness  of  the  training  media. 

In  contrast,  a  decrease  in  the  number  of  personnel  to  be  trained  would  not 
significantly  affect  the  LCC  of  the  training  simulators  and  devices  because  the 
fixed  cost  of  procurement  represents  the  most  significant  portion  of  the  LCC. 
Nevertheless,  without  a  requirement  to  fund  substantii  upgrades  for  the 
simulators  and  devices,  the  cost-effectiveness  of  the  training  system  would 
remain  unchanged  because  the  fixed  investment  in  training  devices  and 
simulators  had  dready  been  made. 

The  cost-effectiveness  may  change,  however,  when  the  training  devices  and 
simulators  require  costly  modifications.  Then,  the  cost  of  the  modifications 
increases  the  LCC  for  the  training  simulators  and  devices.  The  revised  cost 
must  then  be  compared  with  the  revised  LCC  of  the  traditional  media  to 
determine  whether  the  current  media  mix  still  represents  the  most  cost-effective 
training  system. 

Training  Media  for  the  B-2  Training  System.  The  cost-effectiveness 
of  the  B-2  training  system  was  affected  by  both  a  decrease  in  the  number  of 
personnel  to  be  trained  and  an  increase  in  LCC  as  a  result  of  the  cost  of 
modifications  to  the  B-2  air  vehicle  baseline.  The  impact  of  the  change  on 
system  cost-effectiveness  should  have  been  determined  by  analysis.  At  the  time 
that  the  Air  Force  made  decisions  concerning  block  upgrades,  the  estimated  cost 
of  the  simulators  and  devices  on  which  the  media  selection  had  been  made  had 
changed.  For  example,  the  contract  budget  base  for  the  Hughes-Link 
Engineering  and  Manufacturing  Development  contract  was  originally 
$133.5  million.  The  current  contract  budget  base,  incorporating  approved 
program  changes,  is  $273  million;  however,  the  September  1996  estimate  at 
completion  of  the  final  Engineering  and  Manufacturing  Development  contract  is 
$548  million.  The  increase  in  the  contract  budget  base  is  105  percent  and  the 
current  estimate  of  the  final  cost  is  an  increase  of  311  percent  more  than  the 
original  estimate. 
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Whether  the  increases  to  the  cost  of  the  training  system,  either  sepmately  or 
when  combined  with  the  reduction  in  the  LCC  of  the  traditional  m^ia,  would 
have  changed  the  cost-effectiveness  of  the  media  cannot  be  known  with 
certainty.  Nevertheless,  cost  changes  of  that  magnitude  were  likely  to  reduce 
the  cost-effectiveness  of  device-based  training.  By  reevaluating  the 
cost-effectiveness  of  the  B-2  training  system  when  the  B-2  program  was 
reduced,  the  Air  Force  may  have  avoided  unnecessary  costs  to  upgrade  and 
maintain  existing  training  simulators  and  devices.  Instead,  the  Air  Force  might 
have  adopted  alternative  forms  of  training  that  were  more  cost-effective. 

Breakout  Analysis.  The  Air  Force  contracted  directly  with  Hughes-Link  for 
the  Weapon  Systems  Trainer  and  the  Mission  Trainer  and  with  Northrop  for  the 
remainder  of  the  training  system,  including  the  Computerized  Maintenance 
Training  System,  Weapon  Systems  Training  Aid,  Coc]q)it  Procedures  Trainers, 
and  Weapons  Loading  Trainer.  The  Air  Force  spent  74  percent  of  the 
Northrop  contract,  valued  at  $414  million,  for  the  training  system  with  only 
three  subcontractors. 

Given  the  high  level  of  subcontractor  support,  the  Air  Force  may  have  put  funds 
to  better  use  by  separately  contracting  for  components  directly  with  the 
manufacturer  or  supplier  and  furnishing  them  to  Northrop  as  Government- 
furnished  material.  However,  the  Air  Force  did  not  perform  a  breakout  analysis 
when  it  contracted  for  the  training  simulators  and  devices. 

Cost  to  Maintain  Concurrency.  The  user  of  the  B-2  training  system 
established  a  primary  requirement  to  develop  and  deliver  a  concurrent  training 
system  before  deployment  of  the  first  operational  B-2  air  vehicle  and  to  modify 
the  equipment  to  the  latest  block  configuration  before  the  delivery  of  the  first  air 
vehicle  from  each  new  block.  Concurrency  involves  having  the  configuration  of 
the  training  devices  mirror  the  aircraft  configuration.  The  decision  to  require 
that  the  training  system  be  delivered  before  the  air  vehicle  and  at  the  same  time 
be  concurrent  with  an  unbuilt  and  undelivered  air  vehicle  was  a  very  expensive 
decision  given  the  constantly  changing  aircraft  configuration,  that  is,  the 
decision  to  build  only  21  B-2  air  vehicles  in  three  different  initial 
configurations. 

The  B-2  program’s  air  vehicle  delivery  strategy  is  separated  into  three 
increments  of  increasing  hardware  and  software  function^  capability.  The 
increments  were  referred  to  as  Block  10,  Block  20,  and  Block  30  air  vehicles. 
In-line  production  changes  to  aircraft  being  delivered  for  operational  use 
require  the  Air  Force  to  maintain  multiple  aircraft  and  air  crew  training  device 
configurations  until  system  maturity  is  reached.  Therefore,  each  of  the  air 
vehicle  configurations  requires  air  crew  training  devices  with  similar  functional 
training  capability  for  air  crews  to  exploit  the  increasing  capabilities  of  the 
aircraft. 

To  successfully  accomplish  concurrent  development  of  the  air  vehicle  and  the 
training  system,  the  air  vehicle  baseline  should  be  relatively  stable.  The 
baseline  for  the  B-2  air  vehicle,  however,  was  not  stable.  For  example,  the 
Air  Force  released  three  major  Operational  Flight  Program  sets  before  the 
delivery  of  the  first  air  vehicle.  Such  updates  required  time  to  integrate  into  the 
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training  system  and  also  required  some  redesign  of  interface  software. 
Ultimately,  maintaining  concurrency  in  the  development  of  the  air  vehicle  and 
the  training  system  resulted  in  a  cost  of  at  least  $578.3  million. 

Representatives  of  the  B-2  Training  Team  indicated  that  the  number  of  months 
between  the  freeze  date  and  the  delivery  date  of  the  B-2  air  vehicle  was 
decreasing.  Freeze  date  refers  to  the  date  that  the  design  for  the  training 
simulation  or  device  is  fixed.  The  date  is  based  on  the  assumption  that  the 
configuration  for  a  given  air  vehicle  block  is  known  with  sufficient  cer^nty  to 
design  a  training  system  to  mirror  that  configuration.  AcWeving  the 
requirement  of  delivery  of  a  training  system  before  the  first  air  vehicle  of  each 
block  required  freezing  the  design  up  to  24  months  before  the  required  delivery 
date  for  the  training  system.  Specifically,  Northrop  incorporated  air  vehicle 
changes  to  the  B-2  training  system  after  the  freeze  date  to  deliver  a  product  that 
was  more  current.  The  Air  Force  may  have  benefited  by  implementing  a 
systematic  process  to  identify  the  best  time  to  freeze  the  air  vehicle  baseline  to 
develop  the  training  system.  A  basis  to  develop  such  a  process  could  be  the 
process  used  by  the  Air  Force  to  certify  system  readiness  for  dedicated 
operational  test  and  evaluation.  The  test  and  evaluation  process  provides  a 
structured  mechanism  for  identifying  and  reducing  risks  associated  with 
transitioning  from  developmental  test  and  evaluation  to  dedicated  operational 
test  and  evaluation.  A  standard  framework  is  detailed  in  various  templates  that 
contain  historical  information  and  practical  advice  about  how  to  reduce  or 
eliminate  risk.  A  process  based  on  templates  may  be  applicable  in  determining 
when  the  B-2  air  vehicle  baseline  is  stable  enough  to  successfully  develop 
training  simulators  and  devices. 


Cost  Performance  Reports 

Cost  performance  reports  required  by  the  contracts  made  with  Northrop  did  not 
provide  sufficient  detail  to  assist  the  Government  in  managing  the  B-2  training 
system  acquisition.  According  to  the  "Cost/Schedule  Control  Systems  Criteria 
Joint  Implementation  Guide,"  October  1,  1987,  the  contractor  normally 
provides  cost  data  to  the  Government  at  the  third  level  of  the  contract  work 
breakdown  structure  or  higher.  The  Government  only  requir^  Northrop  to 
report  the  costs  for  the  B-2  training  system  at  Level  2.  That  decision  meant  that 
the  training  system  cost  and  price  information  in  the  monthly  "Cost 
Performance  Reports"  and  quarterly  "Contract  Funds  Status  Reports"  for  the 
B-2  training  system  was  reported  only  as  a  single  line  item. 

For  the  B-2  cost  performance  reports,  Level  1  was  the  overall  program;  Level  2 
included  9  categories,  for  example,  Test  and  Evaluation,  Air  Vehicle,  and 
Training;  and  Level  3  included  categories  such  as  Instructional  Systems 
Development,  Academic  Training  Materials,  and  Training  Equipment.  The 
report  did  not  follow  the  contract  work  breakdown  structure  detail  that  included 
9  Level  3  categories  that  were  further  enumerated  into  21  Level  4  categories. 
The  contract  work  breakdown  structure  at  ^vel  4  included  a  contract  work 
breakdown  structure  designator  for  each  distinct  device  developed  for  the  B-2 
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training  system.  Northrop  collected  information  as  contract  work  breakdown 
structure  levels  below  Level  2,  but  was  not  contractually  obligated  to  provide 
the  data  to  the  Government. 

Because  the  contract  did  not  require  reporting  of  costs  and  funds  management 
below  Level  2,  the  B-2  Systems  Program  Office  was  unable  to  provide 
definitive  cost  data  about  the  costs  for  the  elements  of  the  Northrop  B-2  training 
system  including  the  cost  of  each  type  of  training  device  and  the  cost  of  the 
block  upgrades.  Cost  information  on  the  cost  of  the  various  devices  and  the 
cost  to  upgrade  the  devices  was  essential  information  for  effective  cost 
management. 

For  the  Hughes-Link  contract,  the  Air  Force  also  required  Level  2  reporting; 
however,  the  impact  of  the  requirement  was  not  the  same  for  the  training 
system.  Level  2  for  the  Northrop  B-2  contract  showed  the  cost  for  the  entire 
B-2  training  system.  Level  2  for  Northrop  was  equivalent  to  Level  1  for  the 
Hughes-Link  contract.  Level  2  for  the  Hughes-Link  contract  included  detail  on 
the  cost  of  the  Weapon  Systems  Trainer  and  Mission  Trainer.  Level  2  for 
Hughes-Link  provid^  significantly  more  cost  information  for  managing  the 
training  system  acquisition  and  was  equivalent  to  Level  3  data  for  the  Northrop 
contract.  Even  so,  those  reports  were  less  than  clear  on  the  reasons  for 
increases  in  the  contract  budget  base  and  the  approved  formal  reprogramming 
adjustments.  Also,  a  representative  of  the  Air  Force  Cost  Analysis  Agency 
indicated  that  failure  to  report  cost  information  at  the  appropriate  level  may 
impact  the  accuracy  of  historical  cost  information  available  for  future  cost 
estimating.  If  program  offices  do  not  appropriately  report  cost  information, 
cost  analysts  will  not  have  the  most  accurate  data  on  which  to  base  future  cost 
estimates. 


Management  Comments  and  Audit  Response 

We  provided  a  draft  of  this  report  on  November  14,  1996.  Although  no 
comments  were  required,  the  Office  of  the  Under  Secretary  of  Defense  for 
Acquisition  and  Technology  (the  Under  Secretary)  respond^  to  our  report. 
The  full  text  of  the  comments  is  in  Enclosure  4. 

Management  Comments.  The  Deputy  Director,  Air  Warfare  Strategic  and 
Tactic^  Systems,  stated  that  the  additional  cost  of  $578.3  million  to  maintain 
concurrency  with  three  block  configurations  was  justified  because  of  the  need  to 
accelerate  operational  use  of  the  B-2  air  vehicle.  Furthermore,  the  Deputy 
Director  indicated  that  comparing  the  cost  of  the  B-2  training  system  to  the  less 
costly  F-22  and  F-15E  training  systems  did  not  recognize  significant  differences 
in  the  training  concepts  and  capabilities.  In  addition,  separately  contracting  for 
components  of  the  B-2  training  system  with  the  equipment  manufacturer  or 
supplier,  instead  of  with  Northrop,  would  have  increased  the  risk  to  the 
Government  for  system  integration  problems  and  cost,  as  well  as  increasing  the 
risk  to  security.  The  Deputy  Director  made  suggestions  to  improve  the  clarity 
and  accuracy  of  the  report. 
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Audit  Response.  The  determination  of  whether  or  not  developing  3  block 
configurations  was  necessary  to  accelerate  operational  use  of  the  21  B-2  air 
vehicles  was  outside  the  scope  of  our  audit.  Nevertheless,  the  decision  resulted 
in  the  need  to  maintain  concurrency  between  the  three  B-2  air  vehicle 
configurations  and  the  training  system.  That  need  to  maintain  concurrency 
between  the  air  vehicle  configurations  and  the  training  system  increased  the  cost 
of  the  training  system  by  about  $27.5  million  per  air  vehicle.  Furthermore,  we 
acknowledge  that  differences  in  training  concepts  and  capabilities  of  the  training 
systems  make  it  difficult  to  compare  training  system  costs.  However,  when 
consider^  from  the  perspective  of  the  number  of  operational  air  vehicles,  the 
B-2  training  system  cost  $69.5  million  per  air  vehicle,  as  compared  to 
$1.7  million  for  the  F-22  and  $1.3  million  for  the  F-15E.  We  continue  to 
question  whether  differences  in  training  concepts  and  capabilities  justify  the 
cost.  In  addition,  the  Services  use  breakout  analyses  to  evaluate  risks  and 
related  costs  and  benefits  of  separately  contracting  for  system  components  with 
equipment  manufacturers  and  suppliers.  If  the  Air  Force  had  conducted  such  a 
breakout  analysis,  it  might  have  had  a  firm  basis  for  its  decision  to  contract  only 
with  Northrop  and  Hughes-Link. 

For  additional  information  on  this  report,  please  contact  Mr.  Raymond  A. 
Spencer,  Audit  Program  Director,  at  (703)  604-9071  (DSN  664-9071)  or 
Mr.  David  F.  Vincent,  Audit  Project  Manager,  at  (703)  604-9058 
(DSN  664-9058).  Enclosure  5  lists  the  distribution  of  this  report.  The  audit 
team  members  are  listed  inside  the  back  cover. 

David  K.  Steensma 
Deputy  Assistant  Inspector  General 
for  Auditing 
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Audit  Process 


Scope  and  Methodology 

We  reviewed  documentation  relating  to  the  acquisition  of  the  B-2  training 
system  dated  from  November  1981  through  July  1996  to  accomplish  our  audit 
objectives.  We  reviewed  the  B-2  training  system,  estimated  to  cost 
$1,460  million,  and  related  cost  data,  contract  information,  planning 
documentation,  and  system  requirements.  We  also  interviewed  key  personnel 
directly  involved  with  the  program.  Because  Inspector  General,  DoD,  Report 
No.  96-028,  "Implementation  of  the  DoD  Management  Control  Program  for 
Major  Defense  Acquisition  Programs,"  November  28,  1995,  ad^uately 
addressed  management  controls  for  the  B-2  program,  we  did  not  review  the 
management  control  program  as  it  relates  to  our  objectives. 


Audit  Period  and  Standards 

We  conducted  this  economy  and  efficiency  audit  from  November  1995  through 
September  1996  in  accordance  with  auditing  standards  issued  by  the 
Comptroller  General  of  the  United  States,  as  implemented  by  the  Inspector 
General,  DoD.  We  did  not  use  computer-processed  data  or  statistical  sampling 
procedures  for  this  audit. 


Organizations  and  Individuals  Visited  or  Contacted 

We  visited  or  contacted  individuals  and  organizations  within  the  DoD  and  the 
Northrop  Grumman  Corporation.  Further  details  are  available  on  request. 


Enclosure  1 


Summary  of  Prior  Audits  and  Other  Reviews 


Inspector  General,  DoD 

Inspector  General,  DoD,  Report  No.  96-028,  "Implementetion  of  the  DoD 
Management  Control  Program  for  Major  Defense  Acquisition  Programs," 
November  28,  1995,  states  that  the  acquisition  community  had  not  effectively 
integrated  DoD  Management  Control  Program  requirements  into  its 
management  assessment  and  reporting  processes.  With  respect  to  the  B-2,  the 
reporting  manager  at  the  B-2  Program  Office  stated  that  management  control 
objectives  and  techniques  in  DoD  Instruction  5000.2  were  not  being  identified 
and  documented  as  part  of  the  management  control  evaluation.  The  report  made 
no  recommendations  specifically  to  the  B-2  Program  Office. 


Air  Force 

Transition  of  the  B-2  to  the  Active  Force.  The  Air  Force  Audit  Agency 
issued  two  audit  reports  related  to  the  B-2  training  system. 

o  The  Air  Force  Audit  Agency  Installation  Report  No.  26096015, 
"Transition  of  the  B-2  to  the  Active  Air  Force,  509th  Bomb  Wing,  Whiteman 
Air  Force  Base,  Missouri,"  April  1,  1996.  The  report  states  that  actual  and 
forecasted  use  of  air  crew  and  maintenance  simulators  did  not  justify  the  level  of 
contract  logistics  support.  In  addition,  forecasts  indicated  that  expected  use  for 
future  years  did  not  justify  planned  contract  logistics  support  for  those  years. 
The  Commander,  509th  Bomb  Wing,  concurred  with  the  finding,  but  indicated 
that  the  level  of  contractor  support  for  B-2  training  simulators  was  not  the 
responsibility  of  the  509th  Bomb  Wing.  Therefore,  the  information  contained 
in  the  Air  Force  report  will  be  forwarded  to  the  audit  control  point  for  use  in 
evaluating  simulator  forecasting  and  contracting  procedures  at  headquarters, 
Air  Combat  Command,  and  the  System  Program  Office,  Wright-Patterson 
Air  Force  Base,  Ohio. 

0  The  Air  Force  Audit  Agency  Installation  Report  No.  44596035, 
"Management  of  the  B-2  Transition  to  the  Active  Force,  Aeronautical  Systems 
Command,  Wright-Patterson  Air  Force  Base,  Ohio,"  March  15,  1996.  T^e 
report  states  that  the  appropriateness  of  contractor  logistics  support  availability 
levels  could  not  be  determined  because  Air  Combat  Command  personnel  had 
not  developed  long  range  training  schedules  for  maintenance  training  devices 
and  air  crew  trainers  and  simulators.  Therefore,  the  issue  was  provided  to  the 
audit  control  point  for  possible  inclusion  in  the  overall  Air  Force  Evaluation. 


I  ^ 
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Summary  of  Prior  Audits  and  Other  Reviews 


Management  of  the  B-2  Training  System.  The  Air  Force  Audit  Agency 
Report  No.  44594024,  "Management  Review  of  B-2  Trainers  and  Simulators, 
Aeronautical  Systems  Command,  Wright-Patterson  Air  Force  Base,  Ohio," 
February  14,  1994,  states  the  following: 

o  Configuration  status  accounting  systems  did  not  provide  data 
consistent  with  the  configuration  management  plans  and  contract  data 
requirements  lists. 

o  Program  office  personnel  did  not  document  consideration  assessments 
for  late  data  deliveries,  and  personnel  did  not  always  accurately  apply  withholds 
for  trainers  delivered  with  open  discrepancy  reports. 

o  The  financial  manager  did  not  reconcile  cost  performance  report 
summary  information  to  detail^  variance  explanations. 

Management  actions  taken  or  planned  were  responsive  to  the  problems 
identified. 
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Components  of  the  B-2  Training  System 


Type  of  Device 

Description 

Maintenance  Training 

Computerized  Maintenance 

Training  System  (CMTS) 

Provides  three-dimensional  system  operation  and 
fault  isolation  training  on  hydraulics/landing  gear, 
fuel,  propulsion,  integrated  avionics, 
electrical/lighting,  armament,  environmental 
conditioning,  and  flight  control  system.  The 
trainers  incorporate  simulated  hardware,  simulation 
models,  and  courseware. 

Weapon  Systems  Training  Aid 
(WSTA) 

A  single-student,  videodisc-based  device  designed 
for  initial  training.  The  WSTA  provides 
two-dimensional  system  operation  and  fault 
isolation  maintenance  training  on  the  same  systems 
as  the  CMTS. 

Weapons  Loading  Trainer  (WLT) 

A  simulated  aircraft  structure  that  consists  of  a 
computational  system,  an  instructor  station,  and  a 
simulator  flight  crew  station.  The  trainer  is  used 
for  training  and  certifying  Aircraft  Armament 

Systems  Specialists/Technicians  to  perform 
on-aircraft  weapons  loading  and  unloading  tasks. 

Crew  Escape  Systems  Maintenance 
Trainer  (CESMT) 

Replicates  both  the  internal  and  external 
characteristics  of  the  flight  crew  compartment. 

The  trainer  is  used  to  train  maintenance  personnel 
in  safe  arming  and  disarming  of  pyrotechnic 
devices,  inspection  and  testing  of  related  seat 
components,  removal  and  installation  of  seats, 
inspection  and  on-aircraft  maintenance  of  related 
seat  components,  and  safe  removal  and  installation 
of  the  escape  hatch  assemblies.  The  trainer  is  also 
used  as  an  egress  safety  familiarization  trainer. 

Flight  Control  Systems  Maintenance 
Trainer  (FCSMT) 

A  simulated  flight  crew  station  frame  with  an 
integrated  rudder  pedal  assembly,  control  stick 
assembly,  throttle  assemblies,  and  computational 
system.  Used  to  train  the  skills  associated  with  the 
fault  isolation,  removal /replacement,  rigging,  and 
functional  check-out  of  the  flight  control  system 
and  throttle  system  assemblies  within  the  flight 
crew  station. 
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Type  of  Device 

Description 

Air  Crew  Training 

Weapon  Systems  Trainer  (WST) 

A  high  fidelity  full-scale  mockup  of  the  B-2  cockpit 
with  all  systems  completely  active  and  integrated. 

The  WST  provides  air  crew  training  in  a  realistic 
combat  environment  with  sufficient  complexity  to 
develop  necessary  air  crew  coordination  required 
during  nuclear,  conventional,  and  training 
operations. 

Mission  Trainer  (MT) 

A  high  fidelity,  full-scale  mockup,  which  contains 
all  instruments,  indicators,  displays,  associated 
lights,  and  controls  of  the  right  side  of  the  B-2 

Crew  Station.  The  MT  supports  Mission 

Commander  qualification  training. 

Cockpit  Procedures  Trainer  (CPT) 

Provides  drill  and  practice  in  procedural  tasks, 
decisionmaking,  and  problem  solving  for  air  crew 
members.  Flight  crew  procedures  training  is 
provided  on  the  CPT  system  through  the  use  of 
simulation  models,  courseware,  and  simulated 
hardware. 

1  ^ 
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OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 


3000  DEFENSE  PENTAGON 
WASHINGTON  DC  20301*3000 


ACQUUUTION  AND 
TCCHNOLOCY 


2  I  JAN  «07 


subject: 

Mr/a  .  |.s ^ 

beyond  the  expected  range. 

"‘‘"VntJS  "IhU  iigh-lideUtr?Sninnystem"reLces  the  use 
^fSrcrafrfor  SrouSKnd  airborne  training,  lowering  associated 
cLran^risk,  a^strategy  essential  f°r  prudent  management  and 
rLdiness  of  a  small,  high-value  aircraft  fleet. 

The  enclosed  response  provides  comments  and 
corrections!  We  apprLiate  the  opportunity  to  review  the  draft 

report . 


^Ronald  Mut^l^rg 
Deputy  Director  Air  Warfare 
Strategic  and  Tactical  Systems 
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Office  of  the  Under  Secretary  of  Defense  Conunents 


Final  Report 
Reference 


revised 


DoD  IG  Draft  Audit  Report 
(Project  No.  5AB-0070.01) 

"Lessons  Learned  on  the  B-2  Training  System" 


"Audit  Results";  "(4)  The  Air  Force  Increased  the  cost  of  the 
B-2  training  system  by  $578.3  million  to  maintain  concurrency 
between  the  training  system  and  the  three  B-2  air  vehicle 
configurations."  (pg  1) 

Comment;  The  Air  Force  did  not  arbitrarily  increase  the  cost  of 
the  training  system.  The  decision  to  accelerate  fielding  of  the 
B-2  by  developing  it  through  three  block  configurations 
considered  all  costs,  including  the  required  modifications  to  the 
training  system.  Fidelity  of  the  training  system  with  the  weapon 
system  is  an  operational  requirement  driven  by  safety  and  combat 
readiness.  A  more  accurate  statement  is,  "The  cost  of  developing 
the  B-2  through  three  configurations  to  accelerate  operational 
use,  included  $578.3  million  to  maintain  required  concurrency 
between  the  training  system  and  the  B-2  air  vehicle 
configurations . " 


"The  B-2  Training  System  Procurement";  This  paragraph  states 
that  as  part  of  a  1987  contract,  Northrop  Grumman  Corporation  was 
responsible  for  the  Flight  Control  System  Maintenance  Trainer. 

(pg  3) 

Comment:  The  Flight  Control  System  Maintenance  Trainer  was  not 

part  of  the  1987  training  system  contract.  It  was  placed  on 
contract  in  1996  following  extensive  cost-benefit  analysis  and 
requirements  validation. 


"Discussion":  Table  1  and  the  associated  paragraph  state  the 

cost  of  the  B-2  training  system  is  much  greater  than  for  other 
air  vehicle  training  systems  and  provides  a  cost  per  student  for 
the  B-2  as  $1,533,613.45.  The  report  goes  on  to  state  that  while 
no  other  air  vehicle  directly  compares  with  the  B-2,  the  cost  of 
procuring  the  B-2  training  system  appears  extraordinarily 
expensive  for  the  number  of  air  vehicles  supported,  (pg  3/4) 

Comment:  The  report  provides  a  cost  per  student  on  the  B-2,  but 
not  for  the  other  aircraft,  nor  figures  on  the  number  of  students 
for  any  of  the  systems.  It  would  be  useful  for  the  table  to 
display  the  number  of  students  supported  for  each  air  vehicle. 

Raw  figures  comparing  cost  per  aircraft  or  student  imply  an 
apples-to-apples  comparison  and  do  not  provide  insight  into 
significant  differences  in  the  weapon  training  concept  and 
capabilities  of  the  training  systems.  For  example,  the  B-2 
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Weapon  System  Trainer  currently  provides  20  hours  per  month  of 
training  to  each  mission  qualified  crew  (who  fly  only  9  hours  per 
month  in  a  B-2  aircraft)  and  supports  highly  complex  tas)cs,  such 
as  certifying  aircrews  for  terrain  following/ terrain  avoidance  in 
the  Weapon  System  Trainer.  The  B-2  training  system  is  also 
capable  of  certifying  weapons  load  crews  in  the  Weapons  Loading 
Trainers,  meeting  the  demanding  requirements  of  nuclear  weapons 
certification.  The  table  should  also  caveat  the  F-22  costs  as 
pre-production  estimates. 


^^Breakout  Analysis'^;  The  Air  Force  contracted  directly  with 
Hughes-Link  for  the  Weapon  Systems  Trainer  and  the  Mission 
Trainer  and  with  Northrop  for  the  remainder  of  the  training 
system.  Further,  the  Air  Force  may  have  put  funds  to  better  use 
by  separately  contracting  for  components  directly  with  the 
manufacturer  or  supplier  and  furnishing  them  to  Northrop  as 
government-furnished  material,  (pg  8) 

Comment ;  The  decision  to  contract  with  Northrop  for  the  training 
system  (except  the  trainers  contracted  to  Hughes-Link)  considered 
risk,  program  management,  and  security  as  well  as  cost. 

Northrop,  as  prime  contractor,  was  responsible  for  providing  a 
fully-integrated  training  system  under  the  fixed-price  contract. 
Breaking  out  the  training  system  would  require  additional  program 
office  people  to  manage  the  effort  and  would  have  increased  the 
risk  to  the  government  for  system  integration  problems  and  costs. 
Program  security  classification  was  also  a  factor  that  favored 
limiting  the  number  of  contractors  and  facilities  given  access  to 
the  program. 


^*Cost  to  Maintain  Concurrency;  The  last  paragraph  (line  4) 
reads,  "...the  Air  Force  released  three  major  Operational  Flight 
Plans  before  delivery..."  (pg  8) 

Comment ;  The  correct  term  is  "Operational  Flight  Program  sets." 
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